Compact multifunctional system for imaging spectroscopy

ABSTRACT

A method for obtaining spectral imaging data comprises at least the steps of receiving a sample set of data generated by sampling a spectral property of an image of an object in a spatial basis, wherein the sampling of the spectral property of the image of the object comprises providing a Spectral Filter Array (SFA) by arranging a plurality of SFA elements together to form a surface; configuring each SFA element of the plurality of SFA elements to filter one or more spectral bandwidths centered each at specific wavelengths corresponding to that SFA element, whereby all of the plurality of SFA elements taken together cover a determined spectral range; and setting the specific wavelengths of each SFA element of the plurality of SFA elements on the surface such to obtain a uniform and aperiodic spatial distribution of all of the plurality of SFA elements across the surface. The sampling of the spectral property of the image of the object further comprises providing an image sensor configured to record at each pixel the light filtered by one of the plurality of SFA elements or a subset of the plurality of SFA elements thereby producing one intensity value of light filtered by the one of the plurality of elements or the subset of the plurality of SFA elements per pixel; forming the image of the object on the SFA through a lens or group of lenses; and recording for all of the pixels of the image sensor the spectrally filtered intensity values thereby obtaining a 2-dimensional array of the intensity values corresponding to the image of the object. The method for obtaining spectral imaging data further comprises the step of reconstructing a full 3 dimensional spectral data cube of the imaged object from the sampled 2-dimensional array.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a U.S. national stage application ofPCT/IB2015/055614 filed on Jul. 24, 2015 designating the United States,and claims foreign priority to International patent applicationPCT/IB2014/063366 filed on Jul. 24, 2014, the contents of both documentsbeing herewith incorporated by reference in their entirety.

TECHNICAL FIELD OF INVENTION

The invention relates to the field of imaging spectroscopy, and moreprecisely hyperspectral imaging. More specifically, the inventioncomprises a compressive spectral image acquisition and reconstructionsystem.

BACKGROUND OF INVENTION

Imaging Spectroscopy

Imaging spectroscopy is a digital sensing process in which a scene isoptically sampled in two spatial dimensions and in one spectraldimension, producing a three dimensional data cube. Spectral imagescontain significantly larger amount of spectral information than colorphotography, typically comprising tens or hundreds of well definednarrow spectral channels for each individual pixel of the image. Inother words each spatial pixel of the spectral image contains a spectralresponse of the respective point of the imaged surface. In comparison, acolor image is comprised by three loosely defined Red-Green-Bluechannels. Spectral images further distinguish themselves from colorphotography through the rigorously measured radiometric output: whilecolor is perception based and has no absolute units, spectral radianceis a physical measure whose unit is [Wm⁻² μm⁻¹ sr⁻¹]. As a consequence,spectral imaging is often used to determine the chemical and biologicalcomposition of objects, while avoiding the need for physical contact [1,2]. The acquisition of 3D hyperspectral imaging data is difficultbecause of the two-dimensional nature of the imaging sensors.

In order to obtain the spectral data cube with a two dimensional sensor,most spectral imaging cameras use the following three elements:

-   -   1. An optical element, such as a lens: to focus the optical        scene onto an imaging plane.    -   2. A dispersive element, such as a prism or diffraction grid: to        spatially distribute spectral information from the imaging        plane.    -   3. An imaging sensor, such as a CCD or a CMOS: to spatially        sample the dispersed light.

Due to the spectral dispersion, spatial resolution must be compromised.To overcome the resolution loss, many spectral imaging cameras employsome form of scanning:

-   -   Pushbroom, or line-scan, cameras capture a spatio-spectral slice        of the datacube, using the two dimensions of the imaging sensor        as 1D spectral and 1D spatial sampling. Pushbroom sensors        require movement to reconstruct an entire hyperspectral image,        they require spatial scanning.    -   Band sequential cameras capture a spectral slice of the spectral        data cube, using either rotating or tunable spectral filters.        They use the two dimensions of the imaging sensor as 2D spatial        sampling and scan spectrally by sequentially applying different        spectral filters in front of the imaging sensor, they thus        require spectral scanning.    -   Interferometry-based spectral cameras sequentially capture        interferograms of the data cube onto the image sensor. The        interferograms are generated by tuning an interferometer, a        process similar to spectral scanning.

Frame, or snapshot, spectral cameras acquire the entire data cubewithout scanning, severely compromising resolution. State-of-the-artsnapshot spectral cameras are either based on 2D diffraction grids orsmall filter banks, requiring a high resolution sensor and producingvery low resolution spectral data cubes with respect to scanningspectral cameras [2].

Compressive Sensing

Classical signal processing dictates that in order to sample, thenreconstruct a signal without information loss, the signal has to besampled with at least twice the highest frequency it contains. Thisminimal sampling frequency is also called Nyquist frequency. If samplingis done under the Nyquist frequency, the reconstructed signal will showinformation loss and artefacts such as aliases. The large number ofmeasurements required to reconstruct a full hyperspectral data cubeunder the Nyquist constraint is one of the main reasons why allnon-compressive snapshot spectral cameras have low resolution.

Recent research in signal processing and advances in computational powerhave led to many improvements in signal acquisition bandwidth reductionthrough compressive sensing. At the core of compressive sensing lays thehypothesis that if the signal that needs to be acquired is sparse insome mathematical basis, the number of measurements required toreconstruct the signal are given by its sparsity and can thus besignificantly smaller than the number of measurements dictated by theNyquist frequency.

The effective reconstruction of the original signal using thecompressive sensing mathematical models imposes certain requirements onthe underlying signal sampling methodology, in particular uniformity andaperiodicity.

In areas of applications such as magnetic resonance imaging (MRI),compressive sensing has been successfully implemented in order to reducethe amount of required measurements by a factor of between 10 and 20[4], or, with a corresponding compression ratio of between 1/10 and1/20. For color images, the method has been suggested in [5], but thepractical implementation has been limited by the low information gainand high computational cost with respect to classical demosaicing. Incolor photography, the required compression ratios is 1/3 as the imageshave a small quantity of spectral information.

Compressive sensing has also been applied to spectral imaging inmultiple scientific experiments, showing compression ratios of between1/4 and 1/16, [6][7]. These experiments, however, failed to produce theresults needed for a practical snapshot spectral camera which requirescompression ratios around 1/50 or more. The color imaging methodsdescribed in [5] cannot be directly applied to 2D spectralsampling-based imaging for a plurality of reasons:

-   -   Correct pixel-wise spectral reconstruction poses a series of new        challenges such as optimal spectral distribution of filters, or        higher order filter response cancellation.    -   Color filter arrays do not provide sufficient spectral        separation    -   For hyperspectral image reconstruction, the algorithms need to        reconstruct a significantly larger set of data from less than 5%        of its size, as opposed to 33% subsampling rate of color        imaging.

Interferometric Spectral Filter Array

Due to the very stringent requirements of spectral imaging, a spectralfilter array is much more difficult to produce than the color arraysused in commercial color cameras. Color cameras generally employpigment-based filters whose spectral transmission bandwidth is too wideand irregular for the radiometric measurements of spectral imaging. Thenumber of different filters that can be produced in a pigment-basedcolor filter array is also limited by the number of pigments used. Thecost of a color filter array thus greatly increases with the increase inthe number of different filters.

Interferometric filters, such as Fabry-Perot filters, are based on adifferent principle than pigment filters and enjoy a number ofproperties useful for imaging spectroscopy:

-   -   finely tunable central wavelength: the central wavelength is        given by the distance between the reflective surfaces used to        make light interfere with itself. Varying this distance changes        the central wavelength of the filter without significantly        altering the shape of its spectral transmission curve;    -   symmetry around central wavelength: the filter transmission is        locally symmetric with respect to the central wavelength, due to        the natural interferometric process;    -   finely tunable bandwidth: the filter bandwidth is given by the        reflectivity of the surfaces which reflect the light;    -   the production cost of a large number of different wavelength        interferometric filters is much lower than for pigment-based        filters.

While their properties make them an obvious choice for imagingspectroscopy, until recently, interferometric filters were not feasibleto create at the size of a single pixel. Fabry-Perot filters were thusused to produce spectral images by band-sequential scanning, as largeglobal filters, which cover the entire sensor surface [8]. However,recent developments in miniaturization and nanotechnology have led tothe development of interferometric spectral filter arrays with theindividual filters being no larger than a pixel [4].

One of the main disadvantages of interferometric filters is their secondand higher order transmissions. These higher order transmissions of thefilters let light pass not only at the desired central wavelength, butalso at twice that wavelength, three times that wavelength and so on.Interferometric filters can also be configured as multiband filters, dueto the higher order transmissions which include multiple specificwavelengths for which they transmit light. In most cases, the higherorder transmissions are undesired effects and need to be nullified.

SUMMARY OF THE INVENTION

In a first aspect, the invention provides a method for obtainingspectral imaging data. The method comprises at least the steps ofreceiving a sample set of data generated by sampling a spectral propertyof an image of an object in a spatial basis, wherein the sampling of thespectral property of the image of the object comprises providing aSpectral Filter Array (SFA) by arranging a plurality of SFA elementstogether to form a surface; configuring each SFA element of theplurality of SFA elements to filter one or more spectral bandwidthscentered each at specific wavelengths corresponding to that SFA element,whereby all of the plurality of SFA elements taken together cover adetermined spectral range; and setting the specific wavelengths of eachSFA element of the plurality of SFA elements on the surface such toobtain a uniform and aperiodic spatial distribution of all of theplurality of SFA elements across the surface. The sampling of thespectral property of the image of the object further comprises providingan image sensor configured to record at each pixel the light filtered byone of the plurality of SFA elements or a subset of the plurality of SFAelements thereby producing one intensity value of light filtered by theone of the plurality of elements or the subset of the plurality of SFAelements per pixel; forming the image of the object on the SFA through alens or group of lenses; and recording for all of the pixels of theimage sensor the spectrally filtered intensity values thereby obtaininga 2-dimensional array of the intensity values corresponding to the imageof the object. The method for obtaining spectral imaging data furthercomprises the step of reconstructing a full 3 dimensional spectral datacube of the imaged object from the sampled 2-dimensional array.

In a preferred embodiment the step of reconstructing comprises amathematical modelling and computational numerical optimization.

In a further preferred embodiment, the mathematical modelling andcomputational numerical optimization comprises a machine learning methodwhich enables an absence of transmission measurement by the SFA.

In a further preferred embodiment, the mathematical modelling andcomputational numerical optimization comprises at least a convexoptimization method based on transmission measurement of the SFA.

In a further preferred embodiment, the step of reconstructing comprisesproviding measured transmissions of individual SFA elements of the SFA;creating a system design matrix from measured transmissions ofindividual SFA elements and image sensor radiometric calibration; andinferring non sampled spectral information using deconvolution ornon-linear sparse reconstruction methods based on the system designmatrix.

In a further preferred embodiment, the step of setting the specificwavelengths of each SFA element of the plurality of SFA elements on thesurface such to obtain the uniform and aperiodic spatial distribution ofthe specific wavelengths of the SFA is deterministic, thereby comprisingan aperiodic tiling such as Wang or Penrose.

In a further preferred embodiment, the step of setting the specificwavelengths of each SFA element of the plurality of SFA elements on thesurface such to obtain the uniform and aperiodic spatial distribution ofthe specific wavelengths of the SFA further comprises filling the SFAsurface with a repeating pattern, then continuously and randomlyinter-changing the SFA elements until the measured entropy of the SFAcentral wavelengths is sufficiently high.

In a further preferred embodiment, the step of setting the specificwavelengths of each SFA element of the plurality of SFA elements on thesurface such to obtain the uniform and aperiodic spatial distribution ofthe specific wavelengths of the SFA further comprises a random samplingof the SFA elements from a uniform distribution.

In a further preferred embodiment, the step of setting the specificwavelengths of each SFA element of the plurality of SFA elements on thesurface such to obtain the multiband distribution of the specificwavelengths of the SFA further comprises a sampling of multiband SFAelements such that neighboring SFA elements are spectrally orthogonal.

In a second aspect, the invention provides a method for Spectral FilterArray (SFA) element second-degree transmission cancellation through SFAdesign and subsequent response subtraction. The method comprises atleast designing a uniformly and aperiodically distributed SpectralFilter Array (SFA) by arranging a plurality of SFA elements together toform a surface; configuring each SFA element of the plurality of SFAelements to filter a spectral bandwidth centered at a central wavelengthcorresponding to that SFA element, whereby all of the plurality ofelements taken together cover a determined spectral range; and settingthe central wavelength of each SFA element of the plurality of SFAelements on the surface such to obtain a distribution of all the centralwavelengths to be uniform and aperiodic over the surface. The methodfurther comprises identifying the SFA elements with second degreetransmission in the sensitivity range of an image sensor; changing thecentral wavelength of SFA elements neighboring the SFA elements withsecond degree transmission, as to match the central wavelength of thesecond degree transmission; re-arranging the SFA element centralwavelengths which were not changed or do not have a second degreetransmission, as to maintain the uniformity of the SFA centralwavelength distribution; and in a 2-dimensional array of pixel intensityvalues, subtracting the response corresponding to the changedneighboring elements from the response corresponding to the SFA elementswhich have a second degree transmission.

In a third aspect, the invention provides a method for obtaining areal-time monochromatic preview of an imaged object, the imaged objecthaving been obtained using the method of obtaining spectral imaging datadescribed herein above, the method for obtaining the real-timemonochromatic preview comprising designing a uniform aperiodic SpectralFilter Array (SFA) by arranging a plurality of SFA elements together toform a surface, configuring each SFA element of the plurality of SFAelements to filter a spectral bandwidth centered at a central wavelengthcorresponding to that SFA element, whereby all of the plurality of SFAelements taken together cover a determined spectral range, and settingthe central wavelength of each SFA element of the plurality of SFAelements on the surface such to obtain a distribution of all the centralwavelengths to be uniform and aperiodic over the surface; generating asubset of SFA elements by a 2-dimensional periodic selection; replacingthe SFA elements belonging to the above subset with SFA elements ofidentical transmission; and from a 2-dimensional array of pixelintensity values, using a subset of values, corresponding to the subsetof periodic SFA elements with identical transmission, to create a lowerresolution monochromatic image.

In a fourth aspect, the invention provides a method for obtaining areal-time monochromatic preview of an imaged object, the imaged objecthaving been obtained using the method of obtaining spectral imaging datadescribed herein above, the method for obtaining the real-timemonochromatic preview comprising designing a uniform aperiodic SpectralFilter Array (SFA) by arranging a plurality of SFA elements together toform a surface, configuring each SFA element of the plurality of SFAelements to filter multiple spectral bands centered at specificwavelengths corresponding to that SFA element, whereby all of theplurality of SFA elements taken together cover a determined spectralrange, and setting the central wavelength of each SFA element of theplurality of SFA elements on the surface such that neighboring SFAelements are spectrally orthogonal; from a 2-dimensional array of pixelintensity values, spatially interpolating each channel corresponding toan SFA element configuration independently at pixel locations whereother SFA element configurations are present; and averaging theindependently interpolated channels, to create a high resolutionmonochromatic image.

In a fifth aspect, the invention provides a method for generating aspectral 3D-model of an imaged object and increasing the spectralreconstruction quality of an imaged scene, comprising taking multipleimages of the scene, each of the images comprising at least an array ofpixel intensities; using monochromatic previews to optically align theimages through image registration, thus generating the 3D-model of thescene; grouping values from different ones of the arrays of pixelintensities corresponding to different images of the multiple images, ifthe pixels are aligned to a same point in the imaged scene; based ongrouped values, re-modeling a system design matrix corresponding tomultiple arrays of pixel intensities, taken from different locations;and reconstructing the spectral texture of the 3D model based on thecross-image system design matrix.

In a sixth aspect, the invention provides a computer program stored on amemory device, the computer program when executed by a computing devicereading the memory device enabling to reconstruct a 3D-hyperspectralimage from a 2D-spatial-spectral dataset, by implementing the steps ofthe method for obtaining spectral imaging data as described hereinabove.

A corresponding hyperspectral image reconstruction module comprises asingle- or multi-processor computing device, or a computer networkprogrammed to reconstruct a 3D hyperspectral image from a 2D dataset.The underlying algorithmic framework of the hyperspectral imagereconstruction may include deconvolution, non-linear sparsereconstruction, regressive machine learning methods, projection of theacquired spectral samples onto a set of a priori known spectralsignatures, as well as any combination of the herein three methods.Algorithmic methods are also provided for variable spatio-spectralresolution reconstruction of the imaged scene, based on a fixedconfiguration imager.

BRIEF DESCRIPTION OF THE FIGURES

The invention will be better understood in view of the description ofpreferred embodiments and in reference to the drawings, wherein:

FIG. 1A is a schematic overview of the core imaging design and process;

FIG. 1B is a variation of the 2D spectral sampling imager based on aspectral filter array adapter fitted between the main imaging lens andthe imaging sensor;

FIG. 1C is a variation of the 2D spectral sampling imager where thereconstructor is integrated into the imager;

FIG. 2A shows a color preview of an aerial image scene, thecorresponding sensor readout and the color preview of the reconstructed100 band spectral data cube, as computed by a simulation. Four points ofinterest are marked on the reconstructed image;

FIG. 2B-C shows a spectral comparison between the original image scenesampled at 100 bands (marked “Original”) and the reconstructed spectraldata cube (marked “Reconstructed”), for each of the four points markedin FIG. 2;

FIG. 3A shows an example of a uniformly distributed and aperiodic 12×12element spectral filter array pattern given by central wavelength innanometers. 61 possible filters were considered with central wavelengthsbetween 400 and 100 nm, at a step of 10 nm;

FIG. 3B is a variation of FIG. 3A where elements having a non-negligiblesecond degree transmission are always placed in the direct vicinity ofelements filtering at twice their central wavelength. These filter pairsare highlighted in gray;

FIG. 3C is a variation of FIG. 3A where the aperiodic pattern isinterleaved with a periodic pattern of non-filtering elements orelements having a common spectral transmission. The elements of theperiodic pattern are highlighted in gray;

FIG. 4 depicts an image filtering configuration where the spectralfilter array is preceded by an anti-aliasing filter and followed by acolor filter array, along the optical path of the light to the imagingsensor;

FIG. 5A-B contain example plots of the continuous and discretizedspectral transmission of a Fabry-Perot spectral array element with acentral wavelength of 700 nm and a full width at half maximum of 100 nm;and

FIG. 6 shows example plots of transmissions of two Fabry-Perot spectralarray elements designed for multiband filtering. Variations in bothmultiband wavelength number and bandwidth are illustrated. A widebandfilter designed with 6 specific wavelengths (above) and a narrowbandfilter designed with 5 multiband wavelengths (below) are plotted.

DETAILED DESCRIPTION OF THE INVENTION

In this section, we outline the advantages of the presented solutionwith respect to prior art and subsequently focus on the technicaldetails pertaining to three major aspects of the invention, namely:

-   -   1. the camera design and functionality;    -   2. the spectral filter array design; and    -   3. the reconstructor embodiments and preferred implementation.

The present invention improves upon prior art in multiple ways. Withrespect to spatial or spectral scanning spectral imaging systems, thepresent invention overcomes the need to scan, sampling the imaged scenesimultaneously while providing similar resolution. Due to the extremelyshort optical path of the present invention, a much smaller and lighterspectral image acquisition system can be built than those requiringspectrally dispersive elements. The reduced complexity of the opticaland mechanical parts also significantly lowers the production cost whencompared to spectral imaging systems with dispersive elements. Withrespect to state-of-the-art snapshot spectral acquisition systems, thepresent invention generates significantly higher resolution spectraldata cubes. The present invention greatly reduces the amount of storagerequired to acquire a spectral data cube as it is reconstructed from therelatively small 2D set of optical samples that does not exceed thenumber of pixels of the imaging sensor, while the full data cubecontains as many elements as the number of pixels on the sensormultiplied by the number of registered spectral bands. The size ratiobetween the 2D set of samples and the reconstructed hyperspectral datacube can vary depending on the configuration, but will typically bebetween 1/10 and 1/200.

Camera

A core embodiment of the invention comprises an imaging spectroscopycamera system depicted in FIG. 1A, employed for producing a 2D set ofspectral samples 8 and a computational method which is utilized toreconstruct a complete 3D spectral data cube from the aforementioned 2Dset of optical samples. More specifically, the camera comprises anoptical lens 5, or a group of lenses, used for producing an image of theobject of interest 1 on the image plane. The camera further comprises aspectral filter array (SFA) 6 wherein the various spectral components ofthe object's image are transmitted to the image sensor, and the imagesensor array 7 arranged to detect the light transmitted by theindividual SFA elements. The camera also contains an analog-to-digitalsignal converter and a storage medium 9, on which the resultant 2Ddatasets are stored.

The corresponding spectral data cube reconstruction module 3, orreconstructor, comprises a single- or multi-processor computing device,or a computer network programmed to reconstruct a 3D spectral image fromthe 2D set of optical samples.

The underlying algorithmic framework of the spectral imagereconstruction may include deconvolution, non-linear sparsereconstruction, projection of the acquired spectral samples onto a setof a priori known spectral signatures, as well as any combination of theherein described three methods.

The imaging process is as follows:

-   -   The imaged scene is focused by the lens onto the plane of the        SFA.    -   Each element of the SFA lets only certain frequencies of light        pass through it, depending on its transmission curve.    -   Light transmitted by the SFA falls onto the imaging sensor which        generates, for each pixel, an electrical charge in a manner        proportional to the amount of light received by the pixel.    -   The electrical charges in the imaging sensor are discretized and        digitized by an analog-to-digital converter, producing the        sensor readout.    -   The sensor readouts are then stored onto the storage medium.    -   The reconstructor will receive the sensor readouts from the        storage medium and will reconstruct the full 3D spectral image.

In some embodiments, the camera of this invention is based on aninterchangeable lens camera as depicted in FIG. 1B. The spectral filterarray 6 and a refocusing lens 14 are built into an adapter 13 which fitsin between the lens 5 and the imaging sensor 7 of the interchangeablelens camera. The image from the lens is focused on the spectral filterarray, then refocused onto the imaging sensor by the refocusing lens. Inthese embodiments, any interchangeable lens color camera becomes thecore of this invention by using such an adapter 13.

Various embodiments of the main imaging lens 5 include a fixed focaloptical lens, or lens groups, while others feature a variable focal andvariable focus lens.

Some embodiments see the storage medium 9 integrated in the camera whileothers have no storage medium, the images being sent to a computer or acomputer network directly after acquisition and analog-to-digitalconversion.

Spectral Filter Array

Most embodiments of the present invention include a uniformlydistributed aperiodic spectral filter array 6, which we refer to as SFA,composed of individual spectral filters, referred to as elements. Anexample of such an SFA embodiment is shown in FIG. 3A where each elementis represented by its central wavelength. In other implementations, theSFA may also contain non-filtering elements, while preserving anaperiodic distribution of interferometric filters across the majority ofits surface, as shown in FIG. 3C. In the preferred implementation, theSFA is composed of interferometric filters such as Fabry-Perot filters.

The uniform aperiodic distribution may be obtained in several ways:

-   -   deterministically through aperiodic tiling such as Wang or        Penrose tiling.    -   by filling the SFA surface with a repeating pattern, then        continuously and randomly inter-changing the SFA elements until        the measured entropy of the SFA central wavelengths distribution        is sufficiently high.    -   by random sampling the SFA elements from a uniform distribution.

Some embodiments of the invention include SFA designs where the secondand higher degree transmissions of the interferometric spectral filtersare nullified by means of several methods:

-   -   A method for second degree cancellation consists of placing the        SFA 6 over an existing color filter array 11 (such as a Bayer        pattern filter array), matching the individual SFA elements'        transmission so that only the first order response will be        transmitted by the color filter array, as shown in FIG. 4.    -   Another method for second order transmission cancellation is        always placing a SFA element with its transmission equal to that        of the second degree of one of its adjacent elements, if those        filters have 2nd degree transmission in the sensitivity range of        the sensor. The design depicted in FIG. 3B, allows for        subtracting the second degree transmission based on adjacent        filters, prior to reconstruction.

Some embodiments of the invention use interferometric filters withmultiband wavelength transmissions, rather than central wavelengths, asthe elements of the SFA. Examples of these multiband filtertransmissions are shown in FIG. 6. In these embodiments, the spatialuniform aperiodic distribution of the elements is chosen to maximize thespectral orthogonality of the multiband filters. By spectralorthogonality between filters we mean that they respond differently tothe same wavelengths. Maximizing spectral orthogonality translates tochoosing filters which do not correlate spectrally.

In various embodiments, an element at periodic positions in the SFA hasthe same transmission across the entire spectral filter array as shownin FIGS. 4 and 1A. This periodic element can be either aninterferometric spectral filter, a pigment-based color filter, or anon-filtering element transmitting all wavelengths of light. The pixelsunder these periodic elements will be used to create preview images 15of the imaged scene. These images can be used as real-time previews ofthe imaged scene or directly in image registration and image based 3Dreconstruction.

Some embodiments of the invention include a camera where ananti-aliasing filter 12 is placed before the spectral filter array, asdepicted in FIG. 4, for homogenizing the light reaching the differentfilters. The anti-aliasing filter may be comprised by: birefringentfilters extending over more than 2×2 of the sensor elements, a defocusedobjective lens with a matched aperture stop, and/or a degraded imaginglens.

Spectral Cube Reconstruction

In various embodiments of the invention, the reconstructor 3 is either asingle computer, a programmable single or multi-processor 10 device or acomputer network programmed to reconstruct the hyperspectral image froman acquired two-dimensional dataset 8 through non-linear sparsereconstruction. The reconstructor can also be a virtual machine,accessible through the internet and ran on cloud-based computationalservers.

Various embodiments feature the reconstructor integrated in the cameraand producing the hyperspectral image immediately after having acquiredthe two-dimensional set of optical samples. FIG. 1C depicts such anintegrated camera, composed of the imaging optics 5 which focus thelight onto the SFA 6 that is deposited over the imaging sensor 7. Datafrom the imaging sensor 8 is then saved on the storage medium 9 andconverted to spectral data by the reconstructor 3, composed of multipleprocessing units 10.

The mathematical model describing the sampling and discretization of theimaged scene by the hyperspectral imager of this invention is describedby the following linear system:Ax=y,  (1)where A is the system design matrix containing the spectral transmissionof the individual elements of the SFA, y is the sensor readout, and x isthe hyperspectral data cube corresponding to the spatio-spectralproperties of the imaged scene. Vectors x and y are thus serializedversions of the 3D hyperspectral and 2D sensor datasets.

While the SFA elements sample the continuous incoming light spectrum,the design matrix A contains discrete versions of the SFA transmissions,whose spectral resolution will be equal to the number of spectral bandsof the reconstructed data cube. This property allows for thereconstruction of a various number of spectral bands, this number havingonly a lower bound, dictated by the minimum number of bands required toproperly represent the SFA element transmission curves. In FIGS. 5A and5B, the effects of the aforementioned discretization and lower bound aredepicted: while 30 and 10 bands produce an adequate representation ofthe SFA element transmission, 3 bands are insufficient.

Under the hypothesis of local spectral homogeneity of the imaged scene,the number of spectral samples in y can be artificially increased bypixel grouping, resulting in a non-diagonal system design matrix A. Thishypothesis can be further enforced by the use of an anti-aliasing filterplaced in front of the SFA. Each spatial pixel of the image sensorreadout constitutes a single spectral sample of the complete spectralresponse of the corresponding point in the imaged scene. Thereconstruction process may include the grouping of multiple spectralsamples per pixel, which are derived from the spatial pixels in theclose vicinity of the processed pixel.

One method for pixel grouping may be a sliding 2D window, which groupsall spectral samples from pixels falling inside its area into thecentral pixel. If the SFA contains a periodic interleaved pattern,another method for pixel grouping may be guided by the contrastgradients observed in the dense image directly obtained from the pixelsbehind periodic elements. Using this dense image allows for grouping ofpixels which do not span across edges or high contrast areas, thusreducing the possible artefacts produced by pixel grouping.

The variable resolution discrete filter representation and the pixelgrouping methods described above work in unison to allow for theadjustment of the reconstructed hyperspectral data cube's spatial andspectral resolutions. Depending on the desired hyperspectral data cuberesolution desired, only A and y need to be recomputed accordingly,while the hyperspectral imager configuration remains fixed.

Reconstruction of the hyperspectral cube can be achieved by applyingregressive machine learning methods [12] such as neural networks orrandom forests to the sensor readout or pixel-grouped sensor readout.These methods take in an input signal or feature (y) and output anestimation of the original signal (x). Through training with a largenumber of examples, models such as neural networks can learn to un-mixand correct spectral data from the sensor readout.

Another method for reconstructing the hyperspectral data cube x from themeasurements y can be modelled as a convex optimization problem:argmin(∥Ax−y∥ ₂ +c(x))  (2)where the function c(x) is an always positive penalty term, constrainingthe search space for the optimal x based on a priori knowledge of theproperties of x. The function c(x) will have low values for instances ofx corresponding to a priori knowledge, while ∥Ax−y∥₂ will have lowvalues for instances of x fitting the measured data y. The convexoptimisation (2) can thus be interpreted as finding the spectral datacube x which simultaneously best fits the measured data y and the apriori knowledge about its structure c(x).

Variations of the aforementioned reconstruction model include changingthe representation of x to a mathematical basis in which x becomessparse. For instance, basis such as Fourier, direct cosine transform,wavelet or gradient are known to be sparse for natural images [9]. Inthese cases, (2) may become:argmin(∥ASx−y∥ ₂ +λ∥Sx∥ ₁)  (3)where S is the matrix representation of the transformation from thechosen sparse mathematical basis and c(x) is replaced with the L₁ normwhich enforces sparsity on the representation of x, while A controls thestrength of penalization. For hyperspectral data cubes, othertransformations can be envisioned, such as projecting the measured dataonto a known set of spectral vectors, obtained through principalcomponent analysis, for instance.

The preferred implementation of the reconstruction algorithm is based ona version of the fast iterative shrinkage-thresholding algorithm (FISTA)introduced in [10], using the total variation (TV) norm as the penaltyterm. The reconstructed data cube is thus found by minimizing:argmin(∥Ax−y∥ ₂ +λ∥x∥ _(TV))  (4)where ∥x∥_(TV) can either be the 3D TV norm of the entire reconstructedhyperspectral data cube, or the sum of the 2D TV norms of the individualspectral bands of the reconstructed hyperspectral data cube. Using thesum of 2D TV norms rather than the 3D TV norm has the advantage ofindependently processing spectral bands and allowing for parallelizationof the most computationally intensive steps of FISTA.

If the measurements y were produced by a system configuration having ananti-aliasing filter in the optical path, the quality and speed of thereconstruction produced by FISTA can be improved by independentlyblurring the spectral bands of the estimated reconstruction at eachiteration of the algorithm, in an amount proportional to the blurringinduced by the anti-aliasing filter. This process can be parallelized.

A simulation of the imaging and reconstruction process of the inventionhas been used to validate the principles of the present invention, theresults of which are presented in FIGS. 2A and 2B. The targetcompression ratio was 1/100, meaning the hyperspectral data cube wasreconstructed from 100 times fewer measurements than its total number ofelements. A SFA model based on Wang tiling and Fabry-Perot filtertransmissions was employed, containing 100 filters of centralwavelengths between 400 and 1000 nm, and 100 nm full width at halfmaximum (FWHM). The SFA element's transmissions were discretized in 100spectral bands. The SFA also contained a periodic pattern of elementshaving the central wavelength at 700 nm, as shown in FIGS. 3C and 5.Gradient-based pixel grouping was applied, grouping together 4 pixelsfrom a vicinity of 5×5 pixels. FISTA was configured to minimize the sumof 2D TV norms of independent bands in parallel and included theblurring step at each iteration.

The obtained simulated sensor readout as well as a color preview of thehyperspectral data cube are shown in FIG. 2A. Four points of interesthave been marked on the reconstructed image, presenting various spectralproperties as well as being located in areas of various spatial detailacross the imaged scene. The spectral properties of the originalspatio-spectral image scene and the reconstructed hyperspectral datacube are compared in these four points, the results shown in FIGS. 2Band 2C. The spectral resolution is notable, where the reconstructedspectrum exhibits detail at much finer resolution than the FWHM of theSFA elements used to sample the spectrum. If a spectral camera with 100nm wide filters at half-maximum had directly sampled such a spectrum,sharp spectral features such as peaks would have been smoothed out bythese filters through the sampling process. However, in ourreconstructions we clearly see sharp peaks and valleys spanning lessthan 10 bands (or 60 nm), well below the FWHM of the SFA elements usedfor the spectral sampling.

Commercial Applications

Imaging spectroscopy technology has numerous and proven applications inresearch and commercial domains, including agriculture, natural resourcemanagement, mineralogy, medicine and manufacturing among others.Specifically, the disclosed invention facilitates the development of anew class of compact, lightweight and inexpensive spectral imagingsensors. The new sensors are suitable for deployment using smallunmanned aircraft systems, and cater for many of the existingapplications, while facilitating a fundamentally new range of usagescenarios.

In particular, in the context of the agricultural and related industriesthe presented market size estimate is discussed in detail in [11].

Airborne spectral imaging constitutes the single most effective methodof large-scale monitoring and analysis of vegetation with a provencapability in:

-   -   early detection, diagnosis and control of plant diseases;    -   stress detection and growth monitoring;    -   detection and control of invasive species.

Despite the many proven benefits, today's airborne spectral imagingtechnology is too expensive and complex for effective exploitation inthe agricultural and related industries. In particular, none of theexisting solutions has gained any significant traction in commercialfarming applications thus far.

The proposed invention has the potential to make systematic spectralmonitoring of vegetation accessible and highly profitable by reducingthe cost of data acquisition and processing by a factor of 10, or more,while providing up to 10% increase in yield and associated revenues forthe customers.

The available US statistics suggests 100 million hectares of primecroplands that may require systematic monitoring at a rate of 5 times ayear or more. Approximately 10,000 spectral imaging systems are requiredin order to conduct the necessary monitoring. The resultant combinedvalue of hardware of data processing services may be estimated as US$540 million for the USA market. Assuming the size of the global marketto be five times the size of the USA market and taking into accountother fields of application including environmental monitoring,forestry, control of invasive species, etc., results in an estimate forthe total global market for airborne spectral vegetation monitoring ofat least $2 billion.

Summarizing, the invention relates to the field of imaging spectroscopy,which is a method for optical sensing of both spatial and spectral imageproperties. Specifically, the invention describes a system andmethodology to obtain spectral images, also known as spectral data cubesdue to their three-dimensional nature comprised by two spatial and onespectral dimensions.

The described system is structurally simple and is designed tofacilitate the manufacturing of a compact and lightweight spectralimaging camera. The system embodiment includes a lens, a 2D spectralfilter array, a 2D imaging sensor, as well as the storage and the dataprocessing mechanisms required to obtain a spectral image. Preferablythe invention comprises the specific configuration of the spectralfilter array comprised by spectral filters positioned in front of theindividual light-sensitive elements of the image sensor. The inventionfurther describes a computational framework utilized for thereconstruction of a 3D spectral data cube from the 2D dataset providedby the imaging sensor, as well as a real-time generation of a previewimage. Algorithmic methods are also provided for variablespatio-spectral resolution reconstruction of the imaged scene, based ona fixed configuration imager.

REFERENCES

-   [1] Ellis, J., (2001) Searching for oil seeps and oil-impacted soil    with hyperspectral imagery, Earth Observation Magazine.-   [2] Lacar, F. M., et al., (2001) Use of hyperspectral imagery for    mapping grape varieties in the Barossa Valley, South Australia,    Geoscience and remote sensing symposium (IGARSS '01)—IEEE 2001    International, vol. 6 2875-2877 p.-   [3] Tack, K., Lambrechts, A., Haspeslagh, L. (2011) Integrated    circuit for spectral imaging system, Patent No. WO/2011/064403-   [4] Lustig et al, (2007) Sparse MRI: The Application of Compressed    Sensing for Rapid MR Imaging, MRM 58:1182-1195-   [5] Singh, T., Singh, M. (2011) Method and System for Compressive    Color Image Sampling and Reconstruction. Patent no. US20110142339    A1.-   [6] A. Wagadarikar, R. John, R. Willett, and David Brady, (2008)    Single disperser design for coded aperture snapshot spectral    imaging, Appl. Opt. 47, B44-B51.-   [7] Golbabaee, M., Vandergheynst, P., (2012) Hyperspectral image    compressed sensing via low-rank and joint-sparse matrix recovery,    IEEE International Conference on Acoustics, Speech and Signal    Processing (ICASSP), p. 2741-2744-   [8] Heikki, S., Ville-Veikko, A., Altti A., Tapani A., Christer H.,    Uula K., Jussi M., Jyrki O., (2009) Novel miniaturized hyperspectral    sensor for UAV and space applications. Proc. SPIE 7474, Sensors,    Systems, and Next-Generation Satellites XIII, 74741M,    doi:10.1117/12.830284.-   [9] Subhasis, S., (2000) Image compression—from DCT to wavelets: a    review. Crossroads 6, doi:10.1145/331624.331630-   [10] Beck, A., Teboulle, M., (2009) Fast Gradient-Based Algorithms    for Constrained Total Variation Image Denoising and Deblurring    Problems, IEEE Transactions on Image Processing, vol. 18, no. 11,    pp. 2419, 2434, November 2009 doi:10.1109/TIP.2009.2028250-   [11] The Economic Impact of Unmanned Aircraft Systems Integration in    the United States, AUVSI, 2013-   [12] T. M. Mitchell, “Machine Learning”, 1997, ISBN: 0070428077    9780070428072

The invention claimed is:
 1. A method for obtaining spectral imagingdata, comprising the steps of: receiving a sample set of data generatedby sampling a spectral property of an image of an object in a spatialbasis, wherein the sampling of the spectral property of the image of theobject comprises providing a Spectral Filter Array (SFA) by arranging aplurality of SFA elements together to form a surface, configuring eachSFA element of the plurality of SFA elements to filter one or morespectral bandwidths centered each at specific wavelengths correspondingto that SFA element, wherein all of the plurality of SFA elements takentogether cover a determined spectral range, and setting the specificwavelengths of each SFA element of the plurality of SFA elements on thesurface such to obtain a statistically uniform and aperiodic spatialdistribution of all of the plurality of SFA elements across the surface,optimizing for local uniformity and filling the surface with a repeatingpattern of super-pixels, each super-pixel including a contiguous area ofSFA elements including all unique SFA spectral transmissions, andcontinuously and randomly inter-changing the SFA elements inside of arespective super-pixel until a measured entropy of all the plurality ofSFA elements central wavelengths converges to a value indicating areaching of aperiodicity; providing an image sensor configured to recordat each pixel the light filtered by one of the plurality of SFA elementsor a subset of the plurality of SFA elements thereby producing oneintensity value of light filtered by the one of the plurality ofelements or the subset of the plurality of SFA elements per pixel,forming the image of the object on the SFA through a lens or group oflenses, recording for all of the pixels of the image sensor thespectrally filtered intensity values thereby obtaining a 2-dimensionalarray of the intensity values corresponding to the image of the object,reconstructing a full 3 dimensional spectral data cube of the imagedobject from the sampled 2-dimensional array.
 2. The method of claim 1,wherein the step of reconstructing comprises: creating a mathematicalmodel as a simple or convolutional neural network; training theconvolutional neural network on a number of pairs of dense 3 dimensionalspectral data cubes and synthetically generated sparsely and uniformlysampled spectral imaging data, such that the convolutional neuralnetwork learns an interpolation function; and applying the trainedconvolutional neural network on the sparsely and uniformly sampledspectral imaging data.
 3. The method of claim 2, wherein the step ofreconstructing further comprises: a convex optimization method based onmeasurements of the spectral transmission of every pixel from thesparsely and uniformly sampled spectral imaging data.
 4. The method ofclaim 1, wherein the step of reconstructing further comprises: creatingthe mathematical model as a system design matrix from measuredtransmissions of every pixel from the sparsely and uniformly sampledspectral imaging data; and inferring non sampled spectral informationusing deconvolution or non-linear sparse reconstruction methods based onthe system design matrix.
 5. The method of claim 1, wherein the step ofsetting the specific wavelengths of each SFA element of the plurality ofSFA elements on the surface such to obtain the statistically uniform andaperiodic spatial distribution of the specific wavelengths of the SFA isdeterministic, and includes an aperiodic tiling method.
 6. The method ofclaim 1, further comprising the steps of: setting a specific spectralresponse of each SFA element of the plurality of SFA elements on thesurface such to obtain the multiband spectral response distribution ofthe specific wavelengths of the SFA by providing a plurality ofmultiband SFA elements such that neighboring SFA elements are spectrallyorthogonal, providing a plurality of multiband SFA elements such thateach SFA element is sensitive in a number of spectral ranges covering asubset of the desired spectral range, and providing a plurality ofmultiband SFA elements such that the collection of the sensitive rangesof the individual SFA elements cover the entire desired spectral range.7. A method for obtaining a real-time monochromatic preview of an imagedobject, the imaged object having been obtained using the method ofobtaining spectral imaging data of claim 1, the method for obtaining thereal-time monochromatic preview comprising the steps of: designing auniform aperiodic Spectral Filter Array (SFA) by arranging a pluralityof SFA elements together to form a surface, configuring each SFA elementof the plurality of SFA elements to filter a spectral bandwidth centeredat a central wavelength corresponding to that SFA element, whereby allof the plurality of SFA elements taken together cover a determinedspectral range, and setting the central wavelength of each SFA elementof the plurality of SFA elements on the surface such to obtain adistribution of all the central wavelengths to be uniform and aperiodicover the surface, generating a subset of SFA elements by a 2-dimensionalperiodic selection, replacing the SFA elements belonging to the abovesubset with SFA elements of identical transmission, and from a2-dimensional array of pixel intensity values, using a subset of values,corresponding to the subset of periodic SFA elements with identicaltransmission, to create a lower resolution monochromatic image.
 8. Amethod for obtaining a real-time monochromatic preview of an imagedobject, the imaged object having been obtained using the method ofobtaining spectral imaging data of claim 1, the method for obtaining thereal-time monochromatic preview comprising the steps of: designing auniform aperiodic Spectral Filter Array (SFA) by arranging a pluralityof SFA elements together to form a surface, configuring each SFA elementof the plurality of SFA elements to filter multiple spectral bandscentered at specific wavelengths corresponding to that SFA element,whereby all of the plurality of SFA elements taken together cover adetermined spectral range, and setting the central wavelength of eachSFA element of the plurality of SFA elements on the surface such thatneighboring SFA elements are spectrally orthogonal, from a 2-dimensionalarray of pixel intensity values, spatially interpolating each channelcorresponding to an SFA element configuration independently at pixellocations where other SFA element configurations are present, andaveraging the independently interpolated channels, to create a highresolution monochromatic image.
 9. A non-transitory computer readablemedium having a computer-executable program stored thereon, thecomputer-executable program, when executed by a computing device,configured to perform a method according to claim 1 to reconstruct a3D-hyperspectral image from a 2D-spatial-spectral dataset.
 10. A methodfor Spectral Filter Array (SFA) element second-degree transmissioncancellation through SFA design and subsequent response subtractioncomprising at least: designing a uniformly and aperiodically distributedSpectral Filter Array (SFA) by arranging a plurality of SFA elementstogether to form a surface, configuring each SFA element of theplurality of SFA elements to filter a spectral bandwidth centered at acentral wavelength corresponding to that SFA element, whereby all of theplurality of elements taken together cover a determined spectral range,and setting the central wavelength of each SFA element of the pluralityof SFA elements on the surface such to obtain a distribution of all thecentral wavelengths to be uniform and aperiodic over the surface;identifying the SFA elements with second degree transmission in thesensitivity range of an image sensor, changing the central wavelength ofSFA elements neighboring the SFA elements with second degreetransmission, as to match the central wavelength of the second degreetransmission, re-arranging the SFA element central wavelengths whichwere not changed or do not have a second degree transmission, as tomaintain the uniformity of the SFA central wavelength distribution, andin a 2-dimensional array of pixel intensity values, subtracting theresponse corresponding to the changed neighboring elements from theresponse corresponding to the SFA elements which have a second degreetransmission.
 11. A non-transitory computer readable medium having acomputer-executable program stored thereon, the computer-executableprogram, when executed by a computing device, configured to perform amethod according to claim 10 to cancel SFA element second-degreetransmission through SFA design and subsequent response subtraction. 12.A method for generating a spectral 3D-model of an imaged object andincreasing the spectral reconstruction quality of an imaged scene,comprising: taking multiple images of the scene, each of the imagescomprising at least an array of pixel intensities, using monochromaticpreviews to optically align the images through image registration, thusgenerating the 3D-model of the scene, grouping values from differentones of the arrays of pixel intensities corresponding to differentimages of the multiple images, if the pixels are aligned to a same pointin the imaged scene, based on grouped values, re-modeling a systemdesign matrix corresponding to multiple arrays of pixel intensities,taken from different locations, and reconstructing the spectral textureof the 3D model based on the cross-image system design matrix.
 13. Anon-transitory computer readable medium having a computer-executableprogram stored thereon, the computer-executable program, when executedby a computing device, configured to perform a method according to claim12 to generate a spectral 3D-model of an imaged object and increasingthe spectral reconstruction quality of an imaged scene.